Add Ollama and Local Model Support

Summary

This PR adds comprehensive support for using Ollama and other local LLM providers (like LM Studio) with upskill's generate and eval commands. Users can now run skill generation and evaluation entirely locally without requiring API keys from cloud providers.

Motivation

Currently, upskill requires API keys for cloud providers (Anthropic, OpenAI, etc.) to function. This PR enables:

• Privacy: Run entirely locally without sending data to external APIs
• Cost savings: Use free local models instead of paid API calls
• Offline usage: Work without internet connectivity
• Model flexibility: Test with any model supported by Ollama (llama3.2, qwen2.5-coder, etc.)

Changes

New Command-Line Flags

Both generate and eval commands now support:

• --base-url <url>: Custom API endpoint for local models (e.g., http://localhost:11434/v1)
• --provider <name>: API provider (optional, auto-detected as generic when --base-url is provided)

Implementation Details

1. Model Factory Monkey Patch

   • Patches FastAgent's ModelFactory.parse_model_string() to handle unknown model names
   • Falls back to generic provider when GENERIC_BASE_URL environment variable is set
   • Catches ModelConfigError and returns ModelConfig(provider=Provider.GENERIC, model_name=model_string)

2. Environment Variable Configuration

   • Sets GENERIC_BASE_URL to point to local API endpoint
   • Sets GENERIC_API_KEY to "local" (required but unused by Ollama)
   • Sets ANTHROPIC_API_KEY to "dummy" to bypass startup checks

3. Model String Formatting

   • For non-generic providers: prepends provider prefix (e.g., anthropic.claude-3-5-sonnet)
   • For generic provider: passes model name as-is (e.g., llama3.2:latest)
   • Monkey patch handles unknown models automatically

4. Small Model Improvements

   • Updated generation prompt to be more explicit and direct
   • Added code fence stripping for models that wrap output in markdown blocks
   • This helps smaller models generate valid SKILL.md format

5. Skills Directory Support

   • FastAgent now loads skills from ./skills/ directory if it exists
   • Added {{agentSkills}} placeholder to skill_gen agent card

Usage Examples

Generate Skill with Ollama

# Generate skill without evaluation
upskill generate "parse YAML files" \
  --model llama3.2:latest \
  --base-url http://localhost:11434/v1 \
  --no-eval \
  -o ./my-skill

# Generate with explicit provider
upskill generate "document code" \
  --model qwen2.5-coder:7b \
  --provider generic \
  --base-url http://localhost:11434/v1 \
  --no-eval

Evaluate Skill with Local Model

# Evaluate with manual test cases
upskill eval ./skills/my-skill \
  --model qwen2.5-coder:7b \
  --base-url http://localhost:11434/v1 \
  --tests tests.json

# Verbose output
upskill eval ./skills/my-skill \
  --model llama3.2:latest \
  --base-url http://localhost:11434/v1 \
  --tests tests.json \
  -v

Testing

Tested with:

• ✅ llama3.2:latest - Basic generation and evaluation works
• ✅ qwen2.5-coder:7b - Better results, 25-75% success rates on simple tasks
• ✅ Skills loading from ./skills/ directory
• ✅ Code fence stripping for wrapped outputs
• ✅ Generate command with --no-eval
• ✅ Eval command with manual test cases (--tests)

Example Test Results

Hello World Skill with qwen2.5-coder:7b:

• Baseline: 50% success (2/4 tests)
• With Skill: 75% success (3/4 tests)
• Improvement: +25%
• Recommendation: Keep skill ✅

Known Limitations

1. Test Case Generation: The --eval-model flag with automatic test generation may not work due to FastAgent's structured() method not properly respecting model overrides.

   Workaround: Use --no-eval flag during generation, then create test cases manually and run eval separately:

   # Generate without eval
   upskill generate "task" --model llama3.2 --base-url http://localhost:11434/v1 --no-eval
   
   # Create tests.json manually
   
   # Evaluate with manual tests
   upskill eval ./skill --model llama3.2 --base-url http://localhost:11434/v1 --tests tests.json

2. Small Model Quality: Models like llama3.2 (3B parameters) may produce lower quality skills compared to larger cloud models. Recommended to use 7B+ models like qwen2.5-coder:7b for better results.

Backwards Compatibility

All changes are fully backwards compatible:

• Existing commands work exactly as before
• New flags are optional
• Default behavior unchanged
• No breaking changes to API or configuration

Files Changed

• src/upskill/cli.py (+148 lines): Added Ollama support, new flags, environment setup
• src/upskill/generate.py (+36 lines): Code fence stripping, improved prompts, RequestParams support
• src/upskill/agent_cards/skill_gen.md (+2 lines): Added {{agentSkills}} placeholder

Future Improvements

Potential follow-ups (not in this PR):

• Document recommended model sizes for different tasks
• Add model capability detection
• Improve test case generation for local models
• Add progress indicators for long-running generations
• Support for other local providers (Llama.cpp, vLLM, etc.)

Checklist

• Code follows project style guidelines
• All changes are backwards compatible
• Tested with multiple local models (llama3.2, qwen2.5-coder)
• Documentation included in commit message
• No breaking changes
• Error handling for missing Ollama server
• Works with both generate and eval commands